- The paper introduces CrackMeBench, a reproducible benchmark where language-model agents analyze symbol-poor binaries and submit passwords, artifacts, or key generators validated by a private executable oracle.
- The evaluation shows GPT-5.5 achieving 11/12 generated-task successes at pass@3, compared with 7/12 for Claude Opus 4.7 and 5/12 for Kimi K2, while performance drops sharply on harder and public tasks.
- The results indicate that agent failures stem largely from time-consuming exploration, runtime-noise triage, and overfitting rather than incorrect submissions, highlighting the value of hidden inputs and auditable execution-based scoring.
CrackMeBench (2605.10597) formulates educational CrackMe-style binary reverse engineering as an executable-oracle agent evaluation problem. Given only a compiled, symbol-poor executable, a language-model agent must recover the validation logic and produce a password, artifact, or key generator that the original program accepts. The benchmark's central design decision is that prose explanations carry no score: every candidate is executed against a private oracle, which makes success externally verifiable and failures attributable to specific operational categories.
Benchmark design
Each task consists of a compiled binary, public metadata, an explicit tool manifest, a submission schema, and a private oracle running outside the agent-visible workspace. Three submission forms are supported: fixed passwords, binary artifacts validated by the program or an embedded VM, and keygen scripts invoked as /work/solve.py USERNAME against hidden usernames sampled by the oracle. The hidden-username sampling is what distinguishes genuine algorithm recovery from hardcoding a single visible example.
The v0 inventory contains twenty tasks split into twelve generated main-score tasks and eight public calibration tasks. The generated tasks are built from seeded C, Rust, and Go templates and divided into six core tasks (literal rodata recovery, XOR/rotate encodings, affine constraints, 32-bit mixing keygens, benign anti-debugging, and a mini bytecode VM) and six harder tasks (opaque predicates, runtime-decrypted validators, toy RSA-style artifacts, checksum/table reasoning, and Rust/Go runtime noise). The public split draws from crackmes.one contributors and is treated strictly as calibration because public pages may contain comments and writeups that leak solutions; agents receive only sanitized binaries with SHA-256 metadata.
Harness and protocol
All agent shell commands execute in a no-network Linux x86-64 Docker container with three mounts: read-only task data (/task), a writable scratch directory (/work), and read-only harness helpers (/harness). At run start, the host executes list_tools --json inside the container and injects the exact tool list into the prompt, so tool availability comes from the environment rather than model priors. The image includes static triage utilities, radare2, headless Ghidra wrappers, gdb/strace/ltrace/qemu, and Python packages including angr, Z3, capstone, unicorn, and pwntools. Command output is stripped of ANSI control sequences before being returned to the model, a detail the authors show matters because verbose interactive output otherwise consumes context and delays submission.
Every run logs Docker image id, prompt hash, command transcripts, token usage, retry events, and oracle decisions. Invalid JSON, missing files, non-executable keygens, and exceeded submission budgets count as protocol failures. The evaluation uses three models under full API identifiers — GPT-5.5 (Azure OpenAI, xhigh reasoning), Claude Opus 4.7 (adaptive thinking, xhigh effort), and kimi-k2-0711-preview — each with a five-minute wall-clock budget and at most three scored submissions per task.
Results
On the generated main-score split, pass@3 is 11/12 (92%) for GPT-5.5, 7/12 (58%) for Claude Opus 4.7, and 5/12 (42%) for Kimi K2. The harder half separates models sharply: 5/6, 2/6, and 1/6 respectively. On public calibration, all models weaken considerably, solving 3/8, 2/8, and 1/8 within five minutes.
| Split |
Model |
pass@3 |
Avg. time (s) |
Avg. tokens |
Est. cost ($) |
| Generated core |
GPT-5.5 |
6/6 |
65.6 |
55,052 |
0.34 |
| Generated core |
Claude Opus 4.7 |
5/6 |
153.3 |
70,624 |
0.41 |
| Generated core |
Kimi K2 |
4/6 |
143.1 |
147,924 |
0.09 |
| Generated hard |
GPT-5.5 |
5/6 |
151.6 |
240,465 |
1.38 |
| Generated hard |
Claude Opus 4.7 |
2/6 |
262.3 |
123,880 |
0.68 |
| Generated hard |
Kimi K2 |
1/6 |
271.6 |
283,106 |
0.18 |
| Public calibration |
GPT-5.5 |
3/8 |
209.5 |
281,747 |
1.59 |
| Public calibration |
Claude Opus 4.7 |
2/8 |
275.9 |
133,067 |
0.71 |
| Public calibration |
Kimi K2 |
1/8 |
299.4 |
620,591 |
0.38 |
GPT-5.5 solves all six core generated tasks and five of six harder ones, missing only the Rust serial task S11. Claude Opus 4.7 succeeds on the packed-loader and toy-crypto artifact tasks but times out on opaque branches, anti-debug timing, checksum reasoning, and both Rust and Go serial recovery. Kimi K2 solves four core tasks and only the toy-crypto task among the harder set. Notably, pass@1 equals pass@3 for nearly all runs: when a model submits a valid candidate, it is almost always correct on the first attempt, so failures manifest as timeouts rather than wrong submissions.
Trace analysis attributes the gap to operational rather than purely semantic differences. GPT-5.5 more often converts recovered constants into a small validation script and submits quickly, whereas Claude and Kimi spend more of the budget on broad static exploration or runtime-noise triage. Two failure modes recur across models: identifying the validation site but continuing to explore until timeout, and overfitting to a visible example in keygen tasks — the latter detectable only because of the hidden-username oracle. These behaviors would be invisible in explanation-only benchmarks, where a plausible narrative can mask whether a recovered algorithm generalizes.
Limitations and open questions
The paper is explicit about scope. The generated tasks are compact and template-authored; public tasks serve only as calibration due to potential solution leakage. All binaries are Linux x86-64 ELFs, leaving Windows PE, Mach-O, firmware, mobile, multi-binary, and longer interactive workloads unaddressed. The shell-command protocol excludes graphical Ghidra workflows and long-lived analyst state, potentially understating capabilities that depend on interactive inspection. The five-minute budget and three-submission cap deliberately do not approximate expert human sessions on the hardest tasks. Failure labels are diagnostic annotations derived from logs, not causal explanations. Because provider snapshots, pricing, and token accounting change, the reported numbers should be read as a timestamped comparison rather than a permanent ranking. An open question the results raise directly: whether the observed gap between controlled generated performance (up to 92%) and public-task performance (at most 38%) reflects implementation noise, unfamiliar input conventions, or template recognition, and how larger budgets change this separation.
Ethically, the benchmark restricts itself to educational, purpose-built binaries with no network access, no real targets, and no malware objectives, scoring accepted inputs rather than patching or bypasses.
Conclusion
CrackMeBench provides a reproducible, narrowly scoped testbed for measuring agent capability on source-free binary validation, positioned between source-level software engineering benchmarks and broad CTF suites. Its executable-oracle scoring, explicit tool disclosure, and structured failure taxonomy yield auditable comparisons, and the v0 results establish both that frontier agents can solve most controlled CrackMe tasks and that they remain weak on externally authored binaries under tight budgets.
