CrackMeBench Benchmark

• CrackMeBench is a benchmark that evaluates language-model agents’ ability to reverse engineer stripped binaries by recovering hidden validation logic.
• It uses an executable-oracle scoring mechanism within a Dockerized setup, ensuring reproducible and outcome-based assessments of binary analysis tasks.
• The tasks range from simple validations to complex, generated puzzles, leveraging static and dynamic analysis to differentiate model performance.

CrackMeBench is a benchmark designed to evaluate the ability of language-model agents to perform classical binary reverse engineering on educational “CrackMe”-style puzzles. Unlike source-level code repair benchmarks or broad capture-the-flag (CTF) evaluations, CrackMeBench isolates the process in which, given a stripped or symbol-poor executable, an agent must infer its hidden validation logic and produce an input, serial, artifact, or key-generator script accepted by the program. The evaluation is strictly outcome-based, relying on whether candidate submissions pass a hidden validation oracle, and operates within a tightly controlled Dockerized environment with explicit tool access, enabling reproducible measurement of agent performance in the core loop of binary analysis (David et al., 11 May 2026).

1. Concept and Motivation

CrackMeBench frames the classical reverse engineering challenge as a deterministic “executable-oracle” problem for autonomous agents. Given only a Linux x86-64 stripped or symbol-poor ELF binary, agents are tasked to recover embedded logic—typically validation routines—and synthesize data (passwords, serials, artifact files, or keygens) that the binary will accept. The benchmark’s focus is the core reverse engineering workflow: binary triage, static and dynamic analysis, hypothesis generation and testing, and inversion of observed logic. This approach diverges from benchmarks tied to source-level synthesis (e.g., HumanEval, MBPP) or heterogeneous CTF benchmarking (e.g., CyberSecEval, NYU CTF Bench), by isolating invertible executable challenges and removing confounds from other task types or source test suites.

Key distinguishing features include:

• Executable-oracle scoring: Credit is contingent exclusively on candidate submission acceptance by the binary’s private oracle; free-form solution explanations are not scored.
• Controlled scope: Tasks are restricted to non-malicious, educational CrackMes with explicit password or keygen logic. All binaries are local (no remote/network access) and no malware is included.
• Explicit tool disclosure: The command-line environment provides a manifest (JSON) of pre-installed tools—objdump, radare2, Ghidra headless, angr, among others—ensuring agents cannot assume capabilities not listed.
• Reproducible, auditable harness: A Docker container with no network access, standardized shell interface, explicit mounts and logging, and strict limits on wall-clock and submission attempts.

2. Task Structure and Generation

Each benchmark task is a compact binary challenge implementing a deterministic fixed validation routine. Typical motifs across tasks include direct literal checks, XOR or rotate-based encodings, pointer hashing, anti-debugging checks, embedded virtual machine (VM) interpreters, toy cryptographic routines, and username-to-serial mapping algorithms.

CrackMeBench v0 consists of two task groups:

• Eight public calibration CrackMes (P01–P08): Sourced from well-known community educational binaries (e.g., simplistic password checks, C++ string manipulations, VM artifact puzzles, XOR-encoded arguments). These serve as calibration only, due to possible prior availability of public write-ups.
• Twelve generated main-score tasks (S01–S12): Created from C, Rust, and Go templates with seeded randomness, these include “core” tasks (e.g., direct rodata checks, affine constraints, bitvector keygens, timing guards, minimal VM interpreters) and “hard” tasks (e.g., opaque predicate mazes, packed XOR loaders, toy RSA checkers, rolling checksums, and Rust/Go serial validation with runtime noise).

Task generation proceeds via seeded template instantiation for validation logic, user-visible parameters (passwords, constants, usernames) and binary structure. Containers host the binaries (compiled with –O2, stripped) and associated metadata, but withhold all generation seeds and oracle details to prevent information leakage.

3. Execution Environment and Agent Access

All agent interactions occur within a constrained Linux x86-64 Docker container, with network interfaces disabled and three explicit mounts:

1. /task:ro (read-only): Provides the challenge binary, metadata, and hashes.
2. /work:rw: Agent scratch space.
3. /harness:ro: Contains utility scripts (e.g., list_tools, submit) and the tool manifest.

At the beginning of each run, /harness/list_tools --json is executed, and its output is injected into the agent prompt to declare all available utilities. The environment includes triage tools (file, strings, readelf, objdump, xxd), decompilers/disassemblers (radare2, Ghidra headless), debuggers/tracers (gdb, strace, qemu-x86_64), Python reverse-engineering frameworks (angr, Z3, capstone), and multi-language build chains (GCC, Clang, Rust, Go), among others. Shell commands are executed via bash -lc within /work, with outputs stripped of ANSI color codes to preserve context constraints.

Agents must format their solution as a JSON object of the form {"type":..., "value":...} or {"path":...} and submit via /harness/submit. The system’s hidden oracle then evaluates inputs on possibly concealed arguments (e.g., usernames) or state. Explanatory output from the agent carries no score; only oracle acceptance governs task success.

4. Evaluation Protocol and Metrics

Each agent-model–task instance operates under a strict protocol:

• Wall-clock time limit: 300 seconds per task instance.
• Submission budget: Maximum of three scored submissions per task.
• Unlimited exploration: Scripted commands and local tests do not count toward the scored attempt limit.

Scoring metrics are focused on:

• pass@1: Fraction of tasks solved on the first scored submission.
• pass@3: Fraction of tasks solved within up to three scored submissions.

Formally, for $k \in \{1, 3\}$,

$$\text{pass}@k = \frac{\#\text{tasks solved within }k\text{ submissions}}{\text{total \# tasks}}$$

Alongside solutions, the harness records full wall-clock time, total command transcript, detailed tool usage categorization, count of agent-model calls, token usage statistics as reported by the provider (input/output/reasoning tokens), estimated dollar cost (at public API rates), and qualitative failure labels (e.g., “triage failure,” “symbolic-exec setup failure,” “timeout”) for diagnostic purposes.

5. Aggregate Results and Observed Behaviors

Aggregate performance on CrackMeBench v0 demonstrates clear model differentiation, particularly on the generated task split:

Model	Generated pass@3	Public pass@3
GPT-5.5	11/12 (92%)	3/8 (38%)
Claude Opus 4.7	7/12 (58%)	2/8 (25%)
Kimi K2	5/12 (42%)	1/8 (12%)

On the “core” (easier) generated tasks, all models show higher success—GPT-5.5 achieves 6/6 (100%) pass@3, Claude Opus 4.7 5/6 (83%), and Kimi K2 4/6 (67%). In contrast, “hard” tasks produce sharper separation (GPT-5.5: 5/6 [83%], Claude: 2/6 [33%], Kimi: 1/6 [17%]; in this subset, only GPT-5.5 fails a Rust serial, Claude times out or fails symbolic reasoning, and Kimi only solves a toy-crypto challenge).

Successful runs typically employ a static-dynamic analysis loop: rapidly extracting candidate literals with tools such as strings or triaging with decompilers, applying dynamic observation, and then synthesizing scripts for keygen tasks. Tool usage profiling indicates that keygen tasks particularly reward models that extract and reconstruct validation logic, enabling reusable scripts over hardcoded candidates.

Common failure patterns include post-check over-exploration (resulting in timeouts), submission formatting errors, hardcoded solutions that fail generalized (e.g., username-based) validation, and error-prone symbolic execution setups.

Resource consumption varies: GPT-5.5 averages ~148 K tokens/run and $0.86 in cost (109 seconds mean solve time); Claude Opus 4.7 ~97 K tokens ($0.54, 208 s), Kimi K2 ~216 K tokens ($0.14, 207 s). Lower-cost models show reduced reliability, largely attributable to increased error rates and less efficient analytical loops.

6. Significance, Limitations, and Future Prospects

CrackMeBench addresses a gap between code synthesis benchmarks and composite CTF platforms, offering an executable-oracle evaluation focused exclusively on reverse engineering small deterministic binaries. The design provides granular assessment of agent capabilities in triaging stripped binaries, leveraging disassembly and dynamic traces, and inverting logic to construct valid candidates.

Current limitations are defined by the v0 protocol: Linux x86-64 ELF-only binaries, strictly educational (non-malicious) tasks, CLI-only interfaces, a five-minute budget, three-submission ceiling, and deterministic task structure. Potential future extensions include support for alternative binary formats (e.g., Windows PE, Mach-O), firmware and mobile targets, multi-binary workflows, broader puzzle categories, and looser time constraints. The potential introduction of GUI toolchains (such as full Ghidra integration) and rotating seeds could support dynamic leaderboards and longitudinal tracking.

A plausible implication is that as agent models develop advanced static/dynamic and symbolic reasoning capabilities, benchmarks like CrackMeBench will be increasingly important for tracking progress toward autonomous, human-competitive binary analysis, while keeping scope, reproducibility, and explicit tool usage transparent (David et al., 11 May 2026).