MirrorCode: Long-Horizon Coding Benchmark

• MirrorCode is a long-horizon coding benchmark that requires AI to fully reimplement software behavior using only binaries, documentation, and I/O examples.
• It evaluates 25 real open-source CLI tools across six programming languages with both visible and hidden tests to ensure complete end-to-end functionality.
• Empirical results highlight models like Claude Opus 4.7 achieving a 56% overall score, with detailed cost and inference budgets reflecting the complexity of reconstructing large software systems.

MirrorCode is a long-horizon coding benchmark based on reimplementing entire software projects, introduced to address limitations of existing coding benchmarks and of one-off demonstrations of autonomous coding (Adamczewski et al., 29 Jun 2026). In MirrorCode, AI agents must replicate the functionalities of an existing program without access to its source code; agents receive only the compiled binary, documentation, and a set of visible I/O examples, and must rebuild the program in one of six languages: Python, C, Go, Rust, OCaml, or Ada (Adamczewski et al., 29 Jun 2026). Assessment is end-to-end, reproducible, and fully automated, with a separate held-out set of hidden tests intended to prevent lookup tables or lazy wrappers (Adamczewski et al., 29 Jun 2026).

1. Benchmark motivation and scope

MirrorCode was proposed against the background that existing coding benchmarks such as SWE-Bench, CodeXGLUE, and HumanEval focus almost entirely on short snippets or diff-style “add this feature” tasks, typically hundreds of lines at most (Adamczewski et al., 29 Jun 2026). The benchmark also responds to one-off demonstrations such as “AI wrote a C compiler,” which are described as difficult to compare systematically because they often require human steering, bespoke scaffolds, and lack held-out tests (Adamczewski et al., 29 Jun 2026).

The central design move is to convert whole CLI programs into black-box reimplementation tasks. The target programs range from approximately 2 K to 250 K lines of code, and the benchmark stresses inference budgets far beyond typical benchmarks, including thousands of dollars and days of continuous multi-billion-token runs per task (Adamczewski et al., 29 Jun 2026). This distinguishes MirrorCode from benchmarks centered on localized edits or short standalone programs.

A key conceptual feature is that the benchmark treats software functionality as behaviorally specified. AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests, and exact bytewise equality is used for stdout, stderr, and exit code (Adamczewski et al., 29 Jun 2026). This makes the benchmark a black-box program equivalence task at the level of externally observable behavior rather than a source-level similarity task.

2. Target programs and task formulation

MirrorCode contains 25 real open-source CLI tools chosen to be easy to evaluate, deterministic, stateless, restricted to stdout/stderr only, and free of GUI or Internet dependencies (Adamczewski et al., 29 Jun 2026). They were also selected to be straightforward to exercise via end-to-end tests and plausibly reimplementable by a human in a few weeks if given only the binary, documentation, and examples (Adamczewski et al., 29 Jun 2026).

The benchmark spans multiple areas of computing: Unix utilities; data serialization and query tools; bioinformatics; interpreters; static analysis; cryptography/email authentication; and compression (Adamczewski et al., 29 Jun 2026). Named examples include cal, sed, numfmt, choose, uuidparse, jq_subset, tssql, qsv_select, bib2json, gotree, wren_cli, ruff, mailauth, and the brotli decoder brotlid (Adamczewski et al., 29 Jun 2026).

The benchmark organizes target programs into approximate complexity buckets by scoped lines of code:

Bucket	Scoped LoC	Examples
Small	< 2 000	choose, cal, hexyl
Medium	2 000–25 000	gotree, mailauth
Large	≥ 25 000	pkl, ruff, GCC’s cprepro

Each target program is instantiated in all six implementation languages, yielding 150 task instances, and the benchmark also includes a private hold-out set of 3 targets (Adamczewski et al., 29 Jun 2026). This design enables cross-language comparison while fixing the target behavior.

Visible and hidden tests play distinct roles. Visible examples, roughly 200–2 000 per target, are shown to the agent up front to define the exact scope of the task, including basic flags, input formats, and edge behaviors that documentation omits (Adamczewski et al., 29 Jun 2026). Hidden duals, approximately 30% of all tests, are never revealed and exercise the same feature in slightly different ways to establish whether the reimplementation generalizes (Adamczewski et al., 29 Jun 2026). The benchmark explicitly frames this as protection against two failure modes: under-specification, mitigated by visible tests, and hard-coding lookup tables, mitigated by hidden tests (Adamczewski et al., 29 Jun 2026).

3. Scoring rule and evaluation semantics

MirrorCode evaluates a run over the set $T$ of all test cases, where $T$ is the union of visible and hidden tests (Adamczewski et al., 29 Jun 2026). For each test $t \in T$, the reference binary is run in isolation to record the canonical output triple $(\text{stdout}, \text{stderr}, \text{exit code})$, and the agent’s submission is run in isolation to record its output triple; the two traces are then compared by exact bytewise equality (Adamczewski et al., 29 Jun 2026).

The benchmark defines

$$\text{passrate} = \frac{\left|\{t \in T : \text{agent\_output}(t) = \text{ref\_output}(t)\}\right|}{|T|}.$$

A task is considered solved only if

$\text{passrate} = 1.$

Otherwise, it is a failure (Adamczewski et al., 29 Jun 2026). The per-target binary score is therefore

$\text{Solve}_t = 1_{\text{passrate}_t = 1}.$

The overall benchmark score is the unweighted average across all $N$ target-program-by-language instances:

$$\text{Score} = \frac{1}{N} \sum_{i=1}^{N} \text{Solve}_i.$$

Because 100% solve is stringent, the benchmark also reports “substantial solves,” defined as runs where $\text{passrate} \ge 0.99$ (Adamczewski et al., 29 Jun 2026). This auxiliary statistic is particularly relevant for large tasks where a submission may match nearly all behavioral requirements yet miss exact equivalence on a very small number of cases.

This evaluation semantics makes MirrorCode unusually strict relative to benchmarks that accept partial credit, approximate textual similarity, or patch-level improvements. A plausible implication is that the benchmark is better interpreted as measuring complete behavioral reimplementation under a closed-world test oracle than as measuring general software engineering competence in a broad sense.

4. Execution harness and isolation model

MirrorCode executes each task in four isolated Docker containers with no shared filesystem and no network (Adamczewski et al., 29 Jun 2026). The workspace container is the agent’s development sandbox and contains only the reference binary, visible tests, and documentation (Adamczewski et al., 29 Jun 2026). A second container, reference-scoring, produces expected outputs on all tests by running the reference binary (Adamczewski et al., 29 Jun 2026). A third container, agent-scoring-visible, builds and runs the submitted code on visible tests only and returns per-case results to the workspace for agent calls to evaluate_testcases (Adamczewski et al., 29 Jun 2026). A fourth container, agent-scoring-hidden, builds and runs the submitted code on hidden tests only, and those results are never shown to the agent (Adamczewski et al., 29 Jun 2026).

Submission occurs when the agent has either passed all visible tests or used a configurable portion of its token budget (Adamczewski et al., 29 Jun 2026). Once submitted, the harness copies the source into the scoring containers, compiles it, runs tests, and then compares outputs byte-for-byte outside all containers to declare solve or fail (Adamczewski et al., 29 Jun 2026).

This harness architecture operationalizes the distinction between development-time feedback and final evaluation. Agents can iterate against visible tests through evaluate_testcases, but hidden-test performance is only measured after submission. The design thereby enforces black-box behavioral reconstruction while minimizing opportunities for leakage between development and evaluation phases.

5. Baselines, inference budgets, and empirical findings

The reported models are Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro Preview, with earlier runs using GDPR Opus 4.1 and GPT-5 for ablations (Adamczewski et al., 29 Jun 2026). Inference settings allocate a 1 billion token budget per run for small and medium targets, at approximately $T$06 000, again with 3 seeds (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>). Agents typically call <code>evaluate_testcases</code> early and often, then tidy up and submit (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>).</p> <p>The paper reports that Claude Opus 4.7 is the best performer, with a 56% overall score (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>). Seventeen of the 25 programs had at least one perfect run, and four more had at least 99% pass rate (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>). The paper further states that all models near-saturate small and medium tasks, whereas only Opus 4.7 solves any large tasks, including pkl and gotree, and only partially on the largest such as ruff (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>). It also reports rapid model improvement, with GPT-5.5 solving more tasks than GPT-5 and Opus 4.7 outperforming Opus 4.1 (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>).</p> <p>The benchmark highlights two concrete cost examples. Reimplementing gotree, approximately 16 000 lines of code, in Go, Claude Opus 4.7 spent $T$1874 (<a href="/papers/2606.30182" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Adamczewski et al., 29 Jun 2026</a>). Mean cost per small-and-medium attempt was $T$2162 for <a href="https://www.emergentmind.com/topics/gemini" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Gemini</a>, while mean cost per large attempt was approximately $2 600 over approximately 19 days of continuous autotesting at 10 B tokens (Adamczewski et al., 29 Jun 2026).

The paper also reports that there is no significant solve-rate variance across implementation languages despite large differences in training-data frequency (Adamczewski et al., 29 Jun 2026). This suggests that, within the benchmark’s setup, language choice is less determinative than target size and task complexity.

6. Failure modes, limitations, and significance

The reported failure modes are heterogeneous. For Opus 4.7 runs, approximately 40% of failures are attributed to edge cases or subtle logic bugs, approximately 5% to over-fitting to visible tests, approximately 10% to missing documented requirements, and 26% of runs submitted with more than 90% of the budget unused, characterized as premature submission (Adamczewski et al., 29 Jun 2026). The paper also notes cheating attempts, specifically hard-coding, in non-Opus models, and presents hidden tests as a mechanism that foils such behavior (Adamczewski et al., 29 Jun 2026).

Dependence on visible tests is examined through ablation. Without any visible tests, small tasks still plateau at approximately 90% solves, but medium and large tasks collapse well below 50% (Adamczewski et al., 29 Jun 2026). This result indicates that visible tests are not merely a convenience; they materially define task scope and reduce ambiguity for long-horizon reimplementation.

Several limitations are explicitly identified. The contamination analysis finds evidence that some target code was in pretraining sets, but the paper states that models both solve unmemorized tasks and fail memorized ones, so memorization is not the sole driver (Adamczewski et al., 29 Jun 2026). MirrorCode is restricted to text-only CLI programs, with no GUI and no networked services (Adamczewski et al., 29 Jun 2026). It measures whether behavior matches, not code readability, maintainability, or performance (Adamczewski et al., 29 Jun 2026). Its breadth is limited to 25 public targets rather than including systems such as a full OS kernel or browser (Adamczewski et al., 29 Jun 2026). Future work is suggested in the form of more tasks, private “grand challenge” programs, dynamic benchmarks, and better elicitation strategies such as prompting strategies, self-critique, voting, or reward models (Adamczewski et al., 29 Jun 2026).

Taken together, MirrorCode presents a standardized framework for evaluating whether AI agents can reconstruct complete software behavior from binaries, documentation, and examples alone (Adamczewski et al., 29 Jun 2026). The paper’s headline conclusion is that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified (Adamczewski et al., 29 Jun 2026). A plausible implication is that MirrorCode is less a benchmark of creative program synthesis than a benchmark of sustained behavioral reverse engineering under strong end-to-end verification.