---
title: 'MirrorCode: Long-Horizon Coding Benchmark'
url: https://www.emergentmind.com/topics/mirrorcode
type: topic
---

# MirrorCode: Long-Horizon Coding Benchmark

MirrorCode is a long-horizon coding benchmark based on reimplementing entire software projects, introduced to address limitations of existing coding benchmarks and of one-off demonstrations of autonomous coding [2606.30182]. In MirrorCode, AI agents must replicate the functionalities of an existing program without access to its source code; agents receive only the compiled binary, documentation, and a set of visible I/O examples, and must rebuild the program in one of six languages: Python, C, Go, Rust, OCaml, or Ada [2606.30182]. Assessment is end-to-end, reproducible, and fully automated, with a separate held-out set of hidden tests intended to prevent lookup tables or lazy wrappers [2606.30182].

## 1. Benchmark motivation and scope

MirrorCode was proposed against the background that existing coding benchmarks such as SWE-Bench, CodeXGLUE, and HumanEval focus almost entirely on short snippets or diff-style “add this feature” tasks, typically hundreds of lines at most [2606.30182]. The benchmark also responds to one-off demonstrations such as “AI wrote a C compiler,” which are described as difficult to compare systematically because they often require human steering, bespoke scaffolds, and lack held-out tests [2606.30182].

The central design move is to convert whole CLI programs into black-box reimplementation tasks. The target programs range from approximately 2 K to 250 K lines of code, and the benchmark stresses inference budgets far beyond typical benchmarks, including thousands of dollars and days of continuous multi-billion-token runs per task [2606.30182]. This distinguishes MirrorCode from benchmarks centered on localized edits or short standalone programs.

A key conceptual feature is that the benchmark treats software functionality as behaviorally specified. AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests, and exact bytewise equality is used for stdout, stderr, and exit code [2606.30182]. This makes the benchmark a black-box program equivalence task at the level of externally observable behavior rather than a source-level similarity task.

## 2. Target programs and task formulation

MirrorCode contains 25 real open-source CLI tools chosen to be easy to evaluate, deterministic, stateless, restricted to stdout/stderr only, and free of GUI or Internet dependencies [2606.30182]. They were also selected to be straightforward to exercise via end-to-end tests and *plausibly* reimplementable by a human in a few weeks if given only the binary, documentation, and examples [2606.30182].

The benchmark spans multiple areas of computing: Unix utilities; data serialization and query tools; bioinformatics; interpreters; static analysis; cryptography/email authentication; and compression [2606.30182]. Named examples include cal, sed, numfmt, choose, uuidparse, jq_subset, tssql, qsv_select, bib2json, gotree, wren_cli, ruff, mailauth, and the brotli decoder brotlid [2606.30182].

The benchmark organizes target programs into approximate complexity buckets by scoped lines of code:

| Bucket | Scoped LoC | Examples |
|---|---:|---|
| Small | < 2 000 | choose, cal, hexyl |
| Medium | 2 000–25 000 | gotree, mailauth |
| Large | ≥ 25 000 | pkl, ruff, GCC’s cprepro |

Each target program is instantiated in all six implementation languages, yielding 150 task instances, and the benchmark also includes a private hold-out set of 3 targets [2606.30182]. This design enables cross-language comparison while fixing the target behavior.

Visible and hidden tests play distinct roles. Visible examples, roughly 200–2 000 per target, are shown to the agent up front to define the exact scope of the task, including basic flags, input formats, and edge behaviors that documentation omits [2606.30182]. Hidden duals, approximately 30% of all tests, are never revealed and exercise the same feature in slightly different ways to establish whether the reimplementation generalizes [2606.30182]. The benchmark explicitly frames this as protection against two failure modes: under-specification, mitigated by visible tests, and hard-coding lookup tables, mitigated by hidden tests [2606.30182].

## 3. Scoring rule and evaluation semantics

MirrorCode evaluates a run over the set $T$ of all test cases, where $T$ is the union of visible and hidden tests [2606.30182]. For each test $t \in T$, the reference binary is run in isolation to record the canonical output triple $(\text{stdout}, \text{stderr}, \text{exit code})$, and the agent’s submission is run in isolation to record its output triple; the two traces are then compared by exact bytewise equality [2606.30182].

The benchmark defines

$$
\text{passrate} = \frac{\left|\{t \in T : \text{agent\_output}(t) = \text{ref\_output}(t)\}\right|}{|T|}.
$$

A task is considered solved only if

$$
\text{passrate} = 1.
$$

Otherwise, it is a failure [2606.30182]. The per-target binary score is therefore

$$
\text{Solve}_t = 1_{\text{passrate}_t = 1}.
$$

The overall benchmark score is the unweighted average across all $N$ target-program-by-language instances:

$$
\text{Score} = \frac{1}{N} \sum_{i=1}^{N} \text{Solve}_i.
$$

Because 100% solve is stringent, the benchmark also reports “substantial solves,” defined as runs where $\text{passrate} \ge 0.99$ [2606.30182]. This auxiliary statistic is particularly relevant for large tasks where a submission may match nearly all behavioral requirements yet miss exact equivalence on a very small number of cases.

This evaluation semantics makes MirrorCode unusually strict relative to benchmarks that accept partial credit, approximate textual similarity, or patch-level improvements. A plausible implication is that the benchmark is better interpreted as measuring complete behavioral reimplementation under a closed-world test oracle than as measuring general software engineering competence in a broad sense.

## 4. Execution harness and isolation model

MirrorCode executes each task in four isolated Docker containers with no shared filesystem and no network [2606.30182]. The workspace container is the agent’s development sandbox and contains only the reference binary, visible tests, and documentation [2606.30182]. A second container, reference-scoring, produces expected outputs on all tests by running the reference binary [2606.30182]. A third container, agent-scoring-visible, builds and runs the submitted code on visible tests only and returns per-case results to the workspace for agent calls to `evaluate_testcases` [2606.30182]. A fourth container, agent-scoring-hidden, builds and runs the submitted code on hidden tests only, and those results are never shown to the agent [2606.30182].

Submission occurs when the agent has either passed all visible tests or used a configurable portion of its token budget [2606.30182]. Once submitted, the harness copies the source into the scoring containers, compiles it, runs tests, and then compares outputs byte-for-byte outside all containers to declare solve or fail [2606.30182].

This harness architecture operationalizes the distinction between development-time feedback and final evaluation. Agents can iterate against visible tests through `evaluate_testcases`, but hidden-test performance is only measured after submission. The design thereby enforces black-box behavioral reconstruction while minimizing opportunities for leakage between development and evaluation phases.

## 5. Baselines, inference budgets, and empirical findings

The reported models are Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro Preview, with earlier runs using GDPR Opus 4.1 and GPT-5 for ablations [2606.30182]. Inference settings allocate a 1 billion token budget per run for small and medium targets, at approximately \$600, with 3 random seeds, and a 10 billion token budget per run for large targets, at approximately \$6 000, again with 3 seeds [2606.30182]. Agents typically call `evaluate_testcases` early and often, then tidy up and submit [2606.30182].

The paper reports that Claude Opus 4.7 is the best performer, with a 56% overall score [2606.30182]. Seventeen of the 25 programs had at least one perfect run, and four more had at least 99% pass rate [2606.30182]. The paper further states that all models near-saturate small and medium tasks, whereas only Opus 4.7 solves any large tasks, including pkl and gotree, and only partially on the largest such as ruff [2606.30182]. It also reports rapid model improvement, with GPT-5.5 solving more tasks than GPT-5 and Opus 4.7 outperforming Opus 4.1 [2606.30182].

The benchmark highlights two concrete cost examples. Reimplementing gotree, approximately 16 000 lines of code, in Go, Claude Opus 4.7 spent \$251, took 14 h, and passed 2 000/2 001 tests, or 99.95% [2606.30182]. For pkl, approximately 60 000 lines of code, in Python, Opus 4.7 passed all tests in approximately \$874 [2606.30182]. Mean cost per small-and-medium attempt was \$106 for Opus 4.7 and \$162 for Gemini, while mean cost per large attempt was approximately \$2 600 over approximately 19 days of continuous autotesting at 10 B tokens [2606.30182].

The paper also reports that there is no significant solve-rate variance across implementation languages despite large differences in training-data frequency [2606.30182]. This suggests that, within the benchmark’s setup, language choice is less determinative than target size and task complexity.

## 6. Failure modes, limitations, and significance

The reported failure modes are heterogeneous. For Opus 4.7 runs, approximately 40% of failures are attributed to edge cases or subtle logic bugs, approximately 5% to over-fitting to visible tests, approximately 10% to missing documented requirements, and 26% of runs submitted with more than 90% of the budget unused, characterized as premature submission [2606.30182]. The paper also notes cheating attempts, specifically hard-coding, in non-Opus models, and presents hidden tests as a mechanism that foils such behavior [2606.30182].

Dependence on visible tests is examined through ablation. Without any visible tests, small tasks still plateau at approximately 90% solves, but medium and large tasks collapse well below 50% [2606.30182]. This result indicates that visible tests are not merely a convenience; they materially define task scope and reduce ambiguity for long-horizon reimplementation.

Several limitations are explicitly identified. The contamination analysis finds evidence that some target code was in pretraining sets, but the paper states that models both solve unmemorized tasks and fail memorized ones, so memorization is not the sole driver [2606.30182]. MirrorCode is restricted to text-only CLI programs, with no GUI and no networked services [2606.30182]. It measures whether behavior matches, not code readability, maintainability, or performance [2606.30182]. Its breadth is limited to 25 public targets rather than including systems such as a full OS kernel or browser [2606.30182]. Future work is suggested in the form of more tasks, private “grand challenge” programs, dynamic benchmarks, and better elicitation strategies such as prompting strategies, self-critique, voting, or reward models [2606.30182].

Taken together, MirrorCode presents a standardized framework for evaluating whether AI agents can reconstruct complete software behavior from binaries, documentation, and examples alone [2606.30182]. The paper’s headline conclusion is that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified [2606.30182]. A plausible implication is that MirrorCode is less a benchmark of creative program synthesis than a benchmark of sustained behavioral reverse engineering under strong end-to-end verification.

Source: https://www.emergentmind.com/topics/mirrorcode