Unix-CTF: Unix Competence Benchmark

• Unix-CTF is a procedural environment generator that isolates Unix competence by generating CTF-style tasks focused on specific Unix features.
• The system employs a five-stage pipeline, including exploration and bidirectional validation, achieving an 87.5% portability rate from 750 test attempts.
• Reinforcement learning experiments improved task-solving performance from 11.6% to 43.6%, highlighting Unix-CTF’s effectiveness in training Unix-specific skills.

Unix-CTF is a procedural environment generator, benchmark, and training surface for Unix competence: the ability to use shell and operating-system primitives as first-class tools rather than merely writing programs through a terminal. It is designed to isolate the Unix component of terminal performance by generating CTF-style tasks in fresh Linux containers, each centered on a single Unix feature, and by validating success through exact flag recovery. In the reported experiments, the resulting skill library yields 155 canonical technique identifiers from 750 raw harvest attempts, and reinforcement learning on this surface raises Qwen3-8B from 11.6% to 43.6% on a 15-skill multi-family holdout (n=225) while also producing a +33 pp gain in Forensics on InterCode-CTF (Bradway et al., 27 May 2026).

1. Conceptual scope and problem formulation

The central claim behind Unix-CTF is that existing terminal benchmarks often conflate two different abilities: programming through a shell and Unix/OS/file-format competence. The authors make that distinction operational by treating Unix competence as skill with shell and operating-system primitives that remain “load-bearing” even when a solver is already strong at Python or general coding. In their taxonomy, the Unix-competence share in benchmarks such as Terminal-Bench 2.0, Cybench, NYU CTF Bench, AgentBench-OS, InterCode-Bash, and Nemotron-Terminal “rarely exceeds 30%,” which motivates a benchmark explicitly targeted at the underrepresented Unix portion (Bradway et al., 27 May 2026).

Unix-CTF therefore frames evaluation around small, mechanically checkable hide-and-recover tasks. Each task hides a short token inside a fresh Linux environment using a single Unix feature or technique, and the agent succeeds only by recovering the exact flag. This design lets the benchmark avoid proxy metrics and tie each instance to a specific Unix surface such as filesystem metadata, process state, binary internals, or service configuration.

The system is described as an offline-harvested, mechanically validated skill library together with an environment generator and a training/evaluation harness. This architecture reflects a deliberate narrowing of scope. Unix-CTF is not presented as a complete substitute for broad cybersecurity benchmarks, live CTF competition, or general terminal coding tasks. Rather, it is meant to isolate the Unix component so that its separability and trainability can be studied directly.

2. Harvest pipeline and bidirectional validation

Unix-CTF’s task generation pipeline has five stages: Explore, Mechanical verify, Synthesize, Portability validate, and Canonicalize. In the Explore stage, Claude Opus 4.7 runs inside a fresh ctf-base Docker container and is instructed to hide a flag of the form flag{\<8--16 hex bytes>} using a target Unix technique, emitting shell commands turn by turn for up to ten turns. Mechanical verify then checks two conditions in that same container: a recursive search for the literal flag must find no occurrences on disk, and the candidate recovery command must print the flag and exit 0. Claude Haiku 4.5 then rewrites successful explorations into parameterized plant.sh(target_directory, flag) and recovery.sh(target_directory) scripts. Portability validate reruns those scripts in a fresh temporary directory with a fresh flag, and Canonicalize greedily deduplicates attempts with difflib.SequenceMatcher at threshold 0.85, preferring fewer dependencies and shorter scripts (Bradway et al., 27 May 2026).

The production funnel is central to the paper’s contribution.

Stage outcome	Count
Raw harvest attempts	750
Survive stage 1	712
Survive stage 2	700
Survive stage 3	693
Survive stage 4	656
Survive stage 5	441
Canonical technique identifiers	155

The authors characterize the core validation rule as a bidirectional contract: no plaintext flag on disk and recovery must work in a fresh directory with a fresh flag. This is stronger than a one-sided solvability check because it requires both non-trivial hiding and reusable recovery. The end-to-end portability rate is 87.5% (656 / 750).

This factorization is also the basis for the paper’s comparison with Endless Terminals. Instead of asking the model to build an entire container from scratch, Unix-CTF lifts the infrastructure into a pre-built base image and asks the model to generate only the planting and recovery logic. In the authors’ reproduction of Endless Terminals with n=120, only 17.5% of tasks were portable, with 71.7% of failures at the Docker-build or test-validation stage and only 10.8% at the LLM task-generation step. This suggests that the bottleneck in prior full-container-generation pipelines is often infrastructure synthesis rather than the invention of hiding techniques themselves (Bradway et al., 27 May 2026).

3. Technique library and Unix surfaces

The harvested library contains 441 deduplicated variants that canonicalize to 155 distinct techniques, with a median of 2 variants per ID and a long tail up to 16 variants. These techniques are organized into 16 post-hoc families: filesystem metadata, binary and ELF internals, shell environment, system services, shell-level cryptographic primitives, processes and IPC, logs, text-processing tools, databases and serialization formats, media and network formats, encoding and steganographic devices, configuration files, users / OS surfaces, math-related tools, archives / compression, and filename tricks and Unicode/stego (Bradway et al., 27 May 2026).

Representative techniques illustrate the intended distinction between Unix competence and generic programming. The paper gives examples including POSIX extended attributes (xattr, getfattr), ELF note sections and build IDs (readelf -n, objcopy, .note.gnu.build-id), pre-epoch modification times (find ! -newermt '1970-01-01'), tmpfs / shared memory (/dev/shm), X.509 custom OIDs and ASN.1 internals, ACLs, file capabilities, sticky/setuid bits, symlink/hardlink tricks, named pipes, Unix sockets, signals, flock, mail spool formats, crontabs, systemd drop-ins, procfs internals such as /proc/<pid>/maps and /proc/<pid>/fd, archive metadata, and Unicode invisibles / zero-width spaces.

The worked exemplars clarify how these surfaces behave in practice. In xattr, the flag is base64-encoded into user.secret on a decoy file and recovered with getfattr --only-values -n user.secret ... | base64 -d. In elf_gnu_build_id, the flag is embedded into the .note.gnu.build-id section of a copied ELF binary and recovered through readelf -n and parsing the build ID. In mtime_pre_epoch, the real file is identified by a timestamp before 1970. In shm_segment, the flag is written into /dev/shm and doubly encoded with base64 + ROT13. In x509_custom_oid, the flag is hidden in a custom X.509 extension and recovered with openssl x509 -text. In whiteout_overlay, one character per directory name is hidden with zero-width spaces.

These examples matter because they target OS-internal and format-internal knowledge that, in the authors’ formulation, lacks a clean Python analogue. A plausible implication is that Unix-CTF is less about shell syntax alone than about the interpretive habits required to inspect Linux state, metadata, binary structure, and service configuration as native objects of reasoning.

4. Environment generation, reward design, and reinforcement learning

Unix-CTF composes library techniques into multi-flag training environments. Given a list of eight techniques and a seed, the environment generator instantiates a pre-built ctf-base container, optionally dresses the filesystem as one of seven server roles—webserver, database, devbox, cicd, mailserver, monitoring, and gateway—samples a target directory from /home/user/, /var/tmp/, /opt/app/, /srv/, or /tmp/, generates a fresh flag for each technique, base64-encodes the technique’s plant.sh into the container and runs it, records exit status and output, and then deletes the plant script so it cannot be recovered from disk. Each environment is a MultiFlagEnv containing the container name, seed, and manifest of planted flags (Bradway et al., 27 May 2026).

The paper uses eight flags per container to densify reward. With only one flag per episode, reward is sparse and binary; with eight flags, each rollout can accumulate partial reward from multiple subgoals. The reward is

$points\_earned - turn\_cost \times turns\_used$

with turn_cost = 1, per-technique points defaulting to 15, each incorrect flag submission costing 1 point, and reward not clipped at zero. The maximum penalty is about -18.

Training uses a sticky-pool scheduler with a rotating pool of 50 techniques. Each environment samples 8 techniques uniformly from the current pool. After each batch, _rotate_pool drops the top rotation_rate × pool_size = 15 pool members by per-technique solve rate, adds one guaranteed random removal for exploration, and replaces them with techniques from the roughly 105 techniques outside the pool, preferring those with the fewest prior attempts. Each batch exposes the model to 64 technique instances via 8 groups × 8 flags.

The reinforcement-learning experiments fine-tune Qwen3-8B with rank-32 LoRA using GRPO. The paper studies three variants: Base Qwen3-8B, GRPO from base init, and GRPO from SFT init. The optional SFT format pass uses 492 successful solver trajectories from an earlier Haiku-solver run and takes about 92 gradient updates. GRPO is run for 40 batches, with 8 groups × 8 rollouts = 64 trajectories per batch, up to 18 turns per rollout, temperature 1.0, KL penalty coefficient 0, and remove_constant_reward_groups enabled; each batch runs on a single n2-highmem-16 GCP instance with eight concurrent Docker containers.

Evaluation is reported on three benchmarks. The internal multi-family holdout uses 15 held-out techniques, 15 trials per technique, total n = 225, one planted flag per environment, and an 18-turn budget. The main results are 11.6% = 26 / 225 for Base, 27.6% = 62 / 225 for GRPO from base init, and 43.6% = 98 / 225 for GRPO from SFT init. On the held-out timeout_race family at n = 30, Base Qwen3-8B achieves 8/30 = 26.7%, CI [14.2%, 44.4%], mean turns 10.5, while GRPO-from-SFT reaches 20/30 = 66.7%, CI [48.8%, 80.8%], mean turns 5.0, with Fisher’s exact two-sided test: p ≈ 0.0040 (Bradway et al., 27 May 2026).

On InterCode-CTF (100 tasks), the totals are 23/100 = 23.0% for Base, 23/100 = 23.0% for GRPO from SFT init, and 32/100 = 32.0% for GRPO from base init. The paper emphasizes that training changes which tasks the model solves more than the aggregate total, with the strongest category-level shift in Forensics: 1/15 = 7% for Base versus 6/15 = 40% for GRPO from base init, a +33 pp gain. On InterCode-Bash fs_1 (60 tasks), the paper reports both threshold ≥ 0.5 and stricter ≥ 0.8, while noting that answer_similarity is identically zero across variants, making the 0.5 threshold misleading; for the Base model, file_diff mean = 0.264, file_changes mean = 0.264, answer_similarity mean = 0.000, and strict success = 0/60.

A notable finding is the ranking inversion between internal and external benchmarks: on the holdout, GRPO-from-SFT is better than GRPO-from-base, but on InterCode-CTF the reverse holds. The appendix further notes that GRPO-from-SFT peaks at about +27.0 batch reward yet reaches only 23% on IC-CTF, whereas GRPO-from-base peaks at only +1.45 batch reward but reaches 32% on IC-CTF. The authors therefore caution that training reward does not reliably predict downstream transfer (Bradway et al., 27 May 2026).

5. Quality audit and comparison with prior procedural generators

Unix-CTF includes a quality audit comparing 439 unix-ctf variants with 120 Endless Terminals task directories, scored by three LLM judges: gpt-5.4-nano, grok-4-20-reasoning, and Kimi-K2.5. The five scored dimensions are input richness, solution non-triviality, format independence, tool surface diversity, and discoverability (Bradway et al., 27 May 2026).

Dimension	unix-ctf	Endless Terminals
Aggregate mean ± std	3.82 ± 0.43	1.88 ± 0.30
Input richness	4.74 ± 0.45	1.81 ± 0.32
Solution non-triviality	3.45 ± 0.34	2.54 ± 0.46
Format independence	4.42 ± 1.00	1.07 ± 0.04
Tool surface diversity	3.44 ± 0.83	2.85 ± 0.78
Discoverability	3.05 ± 0.10	1.12 ± 0.08

The reported headline difference is Δ = +1.94 in aggregate mean score. Judge agreement is also high: Pearson $r$ on per-task aggregate scores is 0.82 for gpt × grok, 0.83 for gpt × kimi, and 0.93 for grok × kimi, with n = 559 common tasks. The audit also reports that 99% of Endless Terminals tasks have at least one hardcoded exact-string assertion, whereas 0% of unix-ctf tasks do.

These measurements are used to support two claims. First, Unix-CTF tasks are richer and less dependent on brittle formatting constraints. Second, the benchmark’s higher score on format independence aligns with the authors’ goal of testing Unix investigation rather than adherence to hidden answer templates. This suggests that the benchmark’s difficulty is intended to arise from Unix surfaces themselves rather than from narrow output-matching artifacts.

6. Position within CTF and agent research

Unix-CTF occupies a specific niche within the broader CTF literature. It does not attempt to solve the same problem as live CTF evaluation. CTFusion argues that static CTF benchmarks are vulnerable to data contamination and cheating, especially when agents are given web search or RAG capability, and proposes a streaming framework built on LIVE CTFS with per-agent independence under a single team account (Lee et al., 12 May 2026). Unix-CTF instead uses procedural generation inside fresh containers. This suggests a complementary relationship: CTFusion addresses benchmark trustworthiness in unreleased live contests, whereas Unix-CTF addresses the direct training and measurement of Unix-specific competence.

Nor is Unix-CTF primarily an agent-interface paper. EnIGMA shows that autonomous CTF solving in Unix-like environments benefits from specialized Interactive Agent Tools, including debugger and server-connection interfaces, rather than a plain one-shot shell (Abramovich et al., 2024). CTFAgent similarly argues that end-to-end CTF solving depends on bridging technical knowledge and environment interaction through two-stage RAG and interactive Environmental Augmentation (Ji et al., 21 Jun 2025). Unix-CTF is compatible with those observations, but its contribution is different: it creates a procedural task surface in which shell, OS, filesystem, and binary-format primitives are themselves the object of training.

The benchmark is also distinct from archival and infrastructure-focused work. CTF Archive preserves hundreds of challenges spanning over a decade in Docker-backed browser environments for sustained education, reducing the burden of rehosting and enabling self-paced study (Gupta et al., 1 Dec 2025). NIZKCTF addresses fairness and auditability by replacing flag submission with public, non-interactive zero-knowledge proofs (Matias et al., 2017). These systems concern persistence, pedagogy, or scoreboard integrity rather than the separability of Unix competence.

A common misconception would be to treat Unix-CTF as a general offensive-security benchmark. The paper argues for a narrower claim: Unix competence is separable from general programming-in-a-terminal and trainable through procedurally generated environments. Broader benchmarks such as CTFTiny include Unix-relevant workflows—ELF reversing, shell spawning, and command-line forensics—but are not Unix-specific in category design (Shao et al., 5 Aug 2025). Unix-CTF’s significance lies precisely in not folding Unix competence back into those broader mixtures. Its primary research claim is that shell and operating-system primitives can be isolated as a distinct capability, trained with reinforcement learning, and evaluated directly rather than inferred indirectly from generic terminal performance (Bradway et al., 27 May 2026).