---
title: 'Unix-CTF: Unix Competence Benchmark'
url: https://www.emergentmind.com/topics/unix-ctf
type: topic
---

# Unix-CTF: Unix Competence Benchmark

Unix-CTF is a procedural environment generator, benchmark, and training surface for **Unix competence**: the ability to use shell and operating-system primitives as first-class tools rather than merely writing programs through a terminal. It is designed to isolate the Unix component of terminal performance by generating CTF-style tasks in fresh Linux containers, each centered on a single Unix feature, and by validating success through exact flag recovery. In the reported experiments, the resulting skill library yields **155 canonical technique identifiers** from **750 raw harvest attempts**, and reinforcement learning on this surface raises Qwen3-8B from **11.6%** to **43.6%** on a **15-skill multi-family holdout (n=225)** while also producing a **+33 pp** gain in **Forensics** on InterCode-CTF [2605.29115].

## 1. Conceptual scope and problem formulation

The central claim behind Unix-CTF is that existing terminal benchmarks often conflate two different abilities: **programming through a shell** and **Unix/OS/file-format competence**. The authors make that distinction operational by treating Unix competence as skill with shell and operating-system primitives that remain “load-bearing” even when a solver is already strong at Python or general coding. In their taxonomy, the Unix-competence share in benchmarks such as **Terminal-Bench 2.0**, **Cybench**, **NYU CTF Bench**, **AgentBench-OS**, **InterCode-Bash**, and **Nemotron-Terminal** “rarely exceeds 30%,” which motivates a benchmark explicitly targeted at the underrepresented Unix portion [2605.29115].

Unix-CTF therefore frames evaluation around small, mechanically checkable hide-and-recover tasks. Each task hides a short token inside a fresh Linux environment using a **single Unix feature or technique**, and the agent succeeds only by recovering the exact flag. This design lets the benchmark avoid proxy metrics and tie each instance to a specific Unix surface such as filesystem metadata, process state, binary internals, or service configuration.

The system is described as an **offline-harvested, mechanically validated skill library** together with an **environment generator** and a **training/evaluation harness**. This architecture reflects a deliberate narrowing of scope. Unix-CTF is not presented as a complete substitute for broad cybersecurity benchmarks, live CTF competition, or general terminal coding tasks. Rather, it is meant to isolate the Unix component so that its separability and trainability can be studied directly.

## 2. Harvest pipeline and bidirectional validation

Unix-CTF’s task generation pipeline has five stages: **Explore**, **Mechanical verify**, **Synthesize**, **Portability validate**, and **Canonicalize**. In the **Explore** stage, **Claude Opus 4.7** runs inside a fresh `ctf-base` Docker container and is instructed to hide a flag of the form `flag{<8--16 hex bytes>}` using a target Unix technique, emitting shell commands turn by turn for up to ten turns. **Mechanical verify** then checks two conditions in that same container: a recursive search for the literal flag must find **no occurrences on disk**, and the candidate recovery command must print the flag and exit **0**. **Claude Haiku 4.5** then rewrites successful explorations into parameterized `plant.sh(target_directory, flag)` and `recovery.sh(target_directory)` scripts. **Portability validate** reruns those scripts in a fresh temporary directory with a fresh flag, and **Canonicalize** greedily deduplicates attempts with `difflib.SequenceMatcher` at threshold **0.85**, preferring fewer dependencies and shorter scripts [2605.29115].

The production funnel is central to the paper’s contribution.

| Stage outcome | Count |
|---|---:|
| Raw harvest attempts | 750 |
| Survive stage 1 | 712 |
| Survive stage 2 | 700 |
| Survive stage 3 | 693 |
| Survive stage 4 | 656 |
| Survive stage 5 | 441 |
| Canonical technique identifiers | 155 |

The authors characterize the core validation rule as a **bidirectional contract**: **no plaintext flag on disk** and **recovery must work in a fresh directory with a fresh flag**. This is stronger than a one-sided solvability check because it requires both non-trivial hiding and reusable recovery. The end-to-end portability rate is **87.5%** (`656 / 750`).

This factorization is also the basis for the paper’s comparison with **Endless Terminals**. Instead of asking the model to build an entire container from scratch, Unix-CTF lifts the infrastructure into a pre-built base image and asks the model to generate only the planting and recovery logic. In the authors’ reproduction of Endless Terminals with `n=120`, only **17.5%** of tasks were portable, with **71.7% of failures** at the Docker-build or test-validation stage and only **10.8%** at the LLM task-generation step. This suggests that the bottleneck in prior full-container-generation pipelines is often infrastructure synthesis rather than the invention of hiding techniques themselves [2605.29115].

## 3. Technique library and Unix surfaces

The harvested library contains **441 deduplicated variants** that canonicalize to **155 distinct techniques**, with a **median of 2 variants per ID** and a long tail up to **16** variants. These techniques are organized into **16 post-hoc families**: **filesystem metadata**, **binary and ELF internals**, **shell environment**, **system services**, **shell-level cryptographic primitives**, **processes and IPC**, **logs**, **text-processing tools**, **databases and serialization formats**, **media and network formats**, **encoding and steganographic devices**, **configuration files**, **users / OS surfaces**, **math-related tools**, **archives / compression**, and **filename tricks and Unicode/stego** [2605.29115].

Representative techniques illustrate the intended distinction between Unix competence and generic programming. The paper gives examples including **POSIX extended attributes** (`xattr`, `getfattr`), **ELF note sections and build IDs** (`readelf -n`, `objcopy`, `.note.gnu.build-id`), **pre-epoch modification times** (`find ! -newermt '1970-01-01'`), **tmpfs / shared memory** (`/dev/shm`), **X.509 custom OIDs and ASN.1 internals**, **ACLs**, **file capabilities**, **sticky/setuid bits**, **symlink/hardlink tricks**, **named pipes**, **Unix sockets**, **signals**, `flock`, **mail spool formats**, **crontabs**, **systemd drop-ins**, **procfs internals** such as `/proc/<pid>/maps` and `/proc/<pid>/fd`, **archive metadata**, and **Unicode invisibles / zero-width spaces**.

The worked exemplars clarify how these surfaces behave in practice. In **xattr**, the flag is base64-encoded into `user.secret` on a decoy file and recovered with `getfattr --only-values -n user.secret ... | base64 -d`. In **elf_gnu_build_id**, the flag is embedded into the `.note.gnu.build-id` section of a copied ELF binary and recovered through `readelf -n` and parsing the build ID. In **mtime_pre_epoch**, the real file is identified by a timestamp before 1970. In **shm_segment**, the flag is written into `/dev/shm` and doubly encoded with base64 + ROT13. In **x509_custom_oid**, the flag is hidden in a custom X.509 extension and recovered with `openssl x509 -text`. In **whiteout_overlay**, one character per directory name is hidden with zero-width spaces.

These examples matter because they target OS-internal and format-internal knowledge that, in the authors’ formulation, lacks a clean Python analogue. A plausible implication is that Unix-CTF is less about shell syntax alone than about the interpretive habits required to inspect Linux state, metadata, binary structure, and service configuration as native objects of reasoning.

## 4. Environment generation, reward design, and reinforcement learning

Unix-CTF composes library techniques into **multi-flag training environments**. Given a list of eight techniques and a seed, the environment generator instantiates a pre-built `ctf-base` container, optionally dresses the filesystem as one of **seven server roles**—**webserver**, **database**, **devbox**, **cicd**, **mailserver**, **monitoring**, and **gateway**—samples a target directory from `/home/user/`, `/var/tmp/`, `/opt/app/`, `/srv/`, or `/tmp/`, generates a fresh flag for each technique, base64-encodes the technique’s `plant.sh` into the container and runs it, records exit status and output, and then deletes the plant script so it cannot be recovered from disk. Each environment is a **MultiFlagEnv** containing the container name, seed, and manifest of planted flags [2605.29115].

The paper uses **eight flags per container** to densify reward. With only one flag per episode, reward is sparse and binary; with eight flags, each rollout can accumulate partial reward from multiple subgoals. The reward is

$$
points\_earned - turn\_cost \times turns\_used
$$

with `turn_cost = 1`, per-technique points defaulting to **15**, each incorrect flag submission costing **1 point**, and reward **not clipped at zero**. The maximum penalty is about **-18**.

Training uses a **sticky-pool scheduler** with a rotating pool of **50 techniques**. Each environment samples **8 techniques uniformly** from the current pool. After each batch, `_rotate_pool` drops the top `rotation_rate × pool_size = 15` pool members by per-technique solve rate, adds **one guaranteed random removal** for exploration, and replaces them with techniques from the roughly **105 techniques outside the pool**, preferring those with the fewest prior attempts. Each batch exposes the model to **64 technique instances** via `8 groups × 8 flags`.

The reinforcement-learning experiments fine-tune **Qwen3-8B** with **rank-32 LoRA** using **GRPO**. The paper studies three variants: **Base Qwen3-8B**, **GRPO from base init**, and **GRPO from SFT init**. The optional SFT format pass uses **492 successful solver trajectories** from an earlier Haiku-solver run and takes about **92 gradient updates**. GRPO is run for **40 batches**, with **8 groups × 8 rollouts = 64 trajectories per batch**, up to **18 turns** per rollout, **temperature 1.0**, **KL penalty coefficient 0**, and `remove_constant_reward_groups` enabled; each batch runs on a single **n2-highmem-16 GCP instance** with eight concurrent Docker containers.

Evaluation is reported on three benchmarks. The internal **multi-family holdout** uses **15 held-out techniques**, **15 trials per technique**, total **n = 225**, one planted flag per environment, and an **18-turn** budget. The main results are **11.6% = 26 / 225** for Base, **27.6% = 62 / 225** for GRPO from base init, and **43.6% = 98 / 225** for GRPO from SFT init. On the held-out **timeout_race** family at **n = 30**, Base Qwen3-8B achieves **8/30 = 26.7%**, CI **[14.2%, 44.4%]**, mean turns **10.5**, while GRPO-from-SFT reaches **20/30 = 66.7%**, CI **[48.8%, 80.8%]**, mean turns **5.0**, with **Fisher’s exact two-sided test: p ≈ 0.0040** [2605.29115].

On **InterCode-CTF (100 tasks)**, the totals are **23/100 = 23.0%** for Base, **23/100 = 23.0%** for GRPO from SFT init, and **32/100 = 32.0%** for GRPO from base init. The paper emphasizes that training changes *which* tasks the model solves more than the aggregate total, with the strongest category-level shift in **Forensics**: **1/15 = 7%** for Base versus **6/15 = 40%** for GRPO from base init, a **+33 pp** gain. On **InterCode-Bash fs_1 (60 tasks)**, the paper reports both threshold **≥ 0.5** and stricter **≥ 0.8**, while noting that `answer_similarity` is identically zero across variants, making the 0.5 threshold misleading; for the Base model, `file_diff` mean = **0.264**, `file_changes` mean = **0.264**, `answer_similarity` mean = **0.000**, and strict success = **0/60**.

A notable finding is the **ranking inversion** between internal and external benchmarks: on the holdout, **GRPO-from-SFT** is better than **GRPO-from-base**, but on InterCode-CTF the reverse holds. The appendix further notes that **GRPO-from-SFT** peaks at about **+27.0** batch reward yet reaches only **23%** on IC-CTF, whereas **GRPO-from-base** peaks at only **+1.45** batch reward but reaches **32%** on IC-CTF. The authors therefore caution that training reward does not reliably predict downstream transfer [2605.29115].

## 5. Quality audit and comparison with prior procedural generators

Unix-CTF includes a quality audit comparing **439 unix-ctf variants** with **120 Endless Terminals task directories**, scored by three LLM judges: **gpt-5.4-nano**, **grok-4-20-reasoning**, and **Kimi-K2.5**. The five scored dimensions are **input richness**, **solution non-triviality**, **format independence**, **tool surface diversity**, and **discoverability** [2605.29115].

| Dimension | unix-ctf | Endless Terminals |
|---|---:|---:|
| Aggregate mean ± std | 3.82 ± 0.43 | 1.88 ± 0.30 |
| Input richness | 4.74 ± 0.45 | 1.81 ± 0.32 |
| Solution non-triviality | 3.45 ± 0.34 | 2.54 ± 0.46 |
| Format independence | 4.42 ± 1.00 | 1.07 ± 0.04 |
| Tool surface diversity | 3.44 ± 0.83 | 2.85 ± 0.78 |
| Discoverability | 3.05 ± 0.10 | 1.12 ± 0.08 |

The reported headline difference is **Δ = +1.94** in aggregate mean score. Judge agreement is also high: Pearson \(r\) on per-task aggregate scores is **0.82** for gpt × grok, **0.83** for gpt × kimi, and **0.93** for grok × kimi, with **n = 559** common tasks. The audit also reports that **99%** of Endless Terminals tasks have at least one hardcoded exact-string assertion, whereas **0%** of unix-ctf tasks do.

These measurements are used to support two claims. First, Unix-CTF tasks are richer and less dependent on brittle formatting constraints. Second, the benchmark’s higher score on **format independence** aligns with the authors’ goal of testing Unix investigation rather than adherence to hidden answer templates. This suggests that the benchmark’s difficulty is intended to arise from Unix surfaces themselves rather than from narrow output-matching artifacts.

## 6. Position within CTF and agent research

Unix-CTF occupies a specific niche within the broader CTF literature. It does not attempt to solve the same problem as **live CTF evaluation**. **CTFusion** argues that static CTF benchmarks are vulnerable to **data contamination** and **cheating**, especially when agents are given web search or RAG capability, and proposes a streaming framework built on **LIVE CTFS** with per-agent independence under a single team account [2605.11504]. Unix-CTF instead uses procedural generation inside fresh containers. This suggests a complementary relationship: CTFusion addresses benchmark trustworthiness in unreleased live contests, whereas Unix-CTF addresses the direct training and measurement of Unix-specific competence.

Nor is Unix-CTF primarily an agent-interface paper. **EnIGMA** shows that autonomous CTF solving in Unix-like environments benefits from specialized **Interactive Agent Tools**, including debugger and server-connection interfaces, rather than a plain one-shot shell [2409.16165]. **CTFAgent** similarly argues that end-to-end CTF solving depends on bridging technical knowledge and environment interaction through **two-stage RAG** and **interactive Environmental Augmentation** [2506.17644]. Unix-CTF is compatible with those observations, but its contribution is different: it creates a procedural task surface in which shell, OS, filesystem, and binary-format primitives are themselves the object of training.

The benchmark is also distinct from archival and infrastructure-focused work. **CTF Archive** preserves **hundreds of challenges spanning over a decade** in Docker-backed browser environments for sustained education, reducing the burden of rehosting and enabling self-paced study [2512.01233]. **NIZKCTF** addresses fairness and auditability by replacing flag submission with public, non-interactive zero-knowledge proofs [1708.05844]. These systems concern persistence, pedagogy, or scoreboard integrity rather than the separability of Unix competence.

A common misconception would be to treat Unix-CTF as a general offensive-security benchmark. The paper argues for a narrower claim: Unix competence is **separable** from general programming-in-a-terminal and **trainable** through procedurally generated environments. Broader benchmarks such as **CTFTiny** include Unix-relevant workflows—ELF reversing, shell spawning, and command-line forensics—but are not Unix-specific in category design [2508.05674]. Unix-CTF’s significance lies precisely in not folding Unix competence back into those broader mixtures. Its primary research claim is that shell and operating-system primitives can be isolated as a distinct capability, trained with reinforcement learning, and evaluated directly rather than inferred indirectly from generic terminal performance [2605.29115].

Source: https://www.emergentmind.com/topics/unix-ctf